Systems and approaches for thermal image corrections

ABSTRACT

A thermal imaging system includes a thermal imaging sensor configured to capture thermal images of assets in an environment, a non-thermal imaging sensor coupled to the thermal sensor and being configured to capture non-thermal images of the assets, an image alignment system, an image correction system, and a controller configured to control operation of the thermal and non-thermal imaging sensors, the image alignment system, and the image correction system. The controller is configured to determine, using the image correction system, that data representative of a reflective surface is present in a present thermal view of a particular asset, cause the thermal and non-thermal imaging sensors to capture pose-corrected images, generate pose corrected thermal and non-thermal images, with at least one having corrected thermal data corresponding to a location of the reflective surface, and associate the corrected thermal data with the at least one reflective surface.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/143,160, filed on Apr. 5, 2015, the contents of which are hereinincorporated by reference in its entirety.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to thermal imaging and, moreparticularly, to systems and approaches for removing image abnormalitiesusing thermal imaging systems.

BACKGROUND

Thermal imaging is a sensing method for non-contact measurement oftemperatures of thermally emissive objects. Thermal imaging devicesdetect radiation emitted by objects by sensing infrared photons andidentifying the flux thereof. By using multiple sensing and focusingelements, the thermal photon flux from separate solid-angular fields ofview can be obtained in a similar manner as visible-light cameras. As aresult, an image of the temperature of objects being captured isgenerated.

Thermal imaging may be used to observe operating characteristics ofelectronic and/or mechanical equipment used in any number of industrialenvironments such as, for example, manufacturing, fabrication, and/orprocessing facilities. For example, localization of objects with heatcontrast can assist in understanding or discerning the location of aconduit in a wall, a source of a water leak, or identifying faultyand/or overloaded circuit breakers. In these examples, a useful imageinterpretation may be straightforward such that an untrained orinexperienced user may perform it and discern the issue. In someexamples, it is advantageous to enhance the interpretability of theimages and perform a thorough analysis of the image and the underlyingdata in order to obtain a binary decision regarding the functional orstructural integrity of an object or the determination of a quantitativefigure relating to the functional or structural integrity thereof. As anexample, in the field of window retrofitting for the purpose ofincreasing energy efficiency, an estimate of the insulation quality of awindow is useful in order to determine the return on investment of areplacement window with a potentially higher insulation quality.

In other examples, the determination of excessive current carriedthrough a circuit breaker can identify a failed closed circuit breakerby determining the breaker rating through optical character recognitionin a visible-light image, integrating the heat associated with thebreaker, and using any number of approaches to calculate the estimatedcurrent carried through the breaker to obtain a binary measure of thebreaker's integrity and/or a probabilistic estimate of the confidence ofthe measure. Thermal imaging may also be incorporated into a predictivemaintenance process used to determine the optimum time when equipmentshould be serviced and/or replaced. Excess heat given off by equipmentis often a key indicator of excess wear or impending equipment failure,thus thermal imaging can serve an integral role in maintaining anefficient and productive work environment.

To perform a thermography scan, i.e., to obtain thermal images ofthermally emissive objects, thermographers first identify all relevantunique objects and/or equipment, commonly referred to as “assets,” whichmay demonstrate an abnormal temperature-related issue. Thermal imagesare taken of each asset using a thermal camera while the thermographeradjusts for contrast and brightness, otherwise known as “level” and“span,” to set a mid-image temperature and temperature ranges in theimage to optimize information presented therein. Setting appropriatelevel and span values is of particular importance because these valuesmust appropriately surround meaningful temperature ranges in order tosee and record abnormalities in thermal images. As an example, if thedesired asset is an electrical panel, there may be a high temperatureconnection that exists under normal operating conditions. To ignore thehigh temperature connection, the top of the temperature range must beadjusted and saturated to allow for other subtle temperature variancesto be seen. As another example, if an abnormally hot connection isidentified, the thermographer may still wish to identify milderabnormally hot conditions. Accordingly, the level and span must beadjusted until any subtler issues become apparent. The thermographer canthen record the image to be reviewed and analyzed. Thermographers willtypically follow a designated route through the environment tosequentially capture images to allow for repeatability on later dates.Upon obtaining images of all desired assets, the thermographer willreview the images and ultimately identify areas of interest and/orconcern.

Typically, this process must be repeated, e.g., every 6 months, tomonitor areas of interest and/or identify any new areas of interest orconcern. However, due to the specialized nature of industrialthermography, thermographers are again needed. Thermographers will thusperiodically return to the environment, as needed, to re-perform theiranalysis. In doing so, they will follow a specified image capture route,created based on the initial thermography scan, that allows them toretrace their steps.

Nonetheless, it will be appreciated that using thermographers to obtainsubsequent images to be analyzed can be prohibitively costly and timeconsuming. As an example, the environment (e.g., the manufacturingfacility) may have to be shut down to perform the thermography scanand/or one or more employees may have to work with the thermographer toperform the scan. Moreover, the acquisition of thermal images of thesame scene at multiple times is typically associated with differences inthe camera location and orientation (or camera “pose”). Thesedifferences in acquisition can result in images that do not match on apixel-by-pixel basis, meaning one pixel does not refer to the same partof the scene or asset within the scene in all images.

Furthermore, in certain environments, assets may have highly reflectivesurfaces. When using a thermal camera, these reflective surfaces arevisible in captured images, and thus may impact image quality and/orcause difficulties in discerning thermal variances in the image. Forexample, the camera operator's hands, face, and body can be sources ofthermal light that are commonly present in reflections. The correctattribution of thermal light to emissive surfaces is complicated whenthe scene contains reflective surfaces because reflective surfaces areeffectively invisible and merely show the thermal photons reflected offof the surface. Thermographers typically ignore the reflections byadjusting their pose and making a mental note to “cancel out” thedifferent patterns of reflections. However, this may result in a poorerquality image since reflections may be present in all perspectives.

BRIEF DESCRIPTION OF THE DRAWINGS

The above needs are at least partially met through provision of thesystems and approaches for thermal image corrections described in thefollowing detailed description, particularly when studied in conjunctionwith the drawings, wherein:

FIG. 1 illustrates a schematic illustration of a thermal imaging devicein accordance with various embodiments of the invention;

FIGS. 2A and 2B illustrate perspective views of an exemplary thermalimaging device of FIG. 1 in accordance with various embodiments of theinvention;

FIG. 3 illustrates a perspective view of the exemplary thermal imagingdevice of FIGS. 1, 2A and 2B, further illustrating fields of view of athermal camera and a non-thermal camera in accordance with variousembodiments of the invention;

FIG. 4 illustrates an exemplary flowchart for calibrating thenon-thermal image sensor and the thermal image sensor of FIGS. 1-3 inaccordance with various embodiments of the invention;

FIG. 5 illustrates an exemplary visualization of the flowchart forcalibrating the non-thermal image sensor and the thermal image sensor ofFIG. 4 in accordance with various embodiments of the invention;

FIG. 6 illustrates an exemplary approach for aligning images taken atdifferent times using an image alignment system in accordance withvarious embodiments of the invention;

FIG. 7 illustrates an exemplary approach for aligning images taken atdifferent times using a feature-based image alignment system inaccordance with various embodiments of the invention;

FIG. 8 illustrates an exemplary approach for aligning images taken atdifferent times using a transform-derived feedback system in accordancewith various embodiments of the invention;

FIG. 9 illustrates an exemplary approach for aligning images using afeature-based image alignment system and an orientation sensor-assistedalignment system in accordance with various embodiments of theinvention;

FIG. 10 illustrates an exemplary approach for identifying the presenceof reflections in images using an image correction system in accordancewith various embodiments of the invention; and

FIG. 11 illustrates an exemplary approach for identifying the presenceof reflective surfaces in images using a plurality of poses along ahemispherical path in accordance with various embodiments of theinvention.

Skilled artisans will appreciate that elements in the figures areillustrated for simplicity and clarity and have not necessarily beendrawn to scale. For example, the dimensions and/or relative positioningof some of the elements in the figures may be exaggerated relative toother elements to help to improve understanding of various embodimentsof the present invention. Also, common but well-understood elements thatare useful or necessary in a commercially feasible embodiment are oftennot depicted in order to facilitate a less obstructed view of thesevarious embodiments. It will further be appreciated that certain actionsand/or steps may be described or depicted in a particular order ofoccurrence while those skilled in the art will understand that suchspecificity with respect to sequence is not actually required. It willalso be understood that the terms and expressions used herein have theordinary technical meaning as is accorded to such terms and expressionsby persons skilled in the technical field as set forth above exceptwhere different specific meanings have otherwise been set forth herein.

DETAILED DESCRIPTION

Generally speaking, pursuant to these various embodiments, systems andapproaches for thermal image corrections are provided. These approachesallow for reflections detected in thermal and non-thermal images to becorrected to allow appropriate thermal data to be provided for locationshaving reflective surfaces. The purpose of identifying thermallyreflective surfaces is to prevent reflected thermal light from beingmisattributed to a reflective surface. This is oftentimes important foraccurate 3-dimensional model estimation and for accurate localization oftemperature measurements.

In many embodiments, a thermal or non-thermal imaging sensor is used toview a particular asset in a present or live view. A reflection in thepresent view is detected using an image correction system. A number ofthermal and non-thermal images of the particular asset are then capturedat different poses or locations surrounding the asset. Next, an imagealignment system determines the homography between the present thermaland/or non-thermal view and each of the captured thermal and/ornon-thermal images to determine whether a suitably low error homographytransform can be performed. In instances in which low error homographytransforms are performed, the images are saved for further analysis.Finally, at relevant pixels of the saved images, a thermal value isobtained that replaces the thermal value or values near the reflectivesurface.

In some embodiments, the image correction system determines the presenceof reflections by identifying an inconsistency between image motion inthe captured thermal and non-thermal images using an image motionestimation technique such as, for example, a feature detection, anoptical flow detection, and the like. A discordance between the presentthermal view and the present non-thermal view may also be used.

The alignment system may determine a homography transform between thepresent thermal view and the captured thermal images. If suitablylow-error homography transforms cannot be performed for a particularimage captured at a particular pose, the image is discarded. Otherwise,this process is repeated for all poses, and a set of pose-correctedimages are saved. It is understood that in some embodiments, the imagealignment system also determines a homography transform between thepresent non-thermal view and the captured non-thermal images.

The image correction system then determines an appropriate thermal valueor values for the portion of the image having a reflective surface. Insome examples, a minimum temperature value determined across the savedthermal images can be used. In other examples, an average or any othervalue may be used.

So configured, a user can determine thermal values for reflectivesurfaces without the presence of patterns from reflected thermal light,thus eliminating the need for the user to perform additionalcalculations and/or impart their own knowledge on the effect reflectivesurfaces may have on thermal data acquisition. Additionally, the systemcan acquire repeatable thermal images of reflective surfaces where thereflected background may vary from one acquisition to another due tochanges in the distribution of thermal sources of reflected lightoriginally emanating from behind the imaging system (such as the cameraoperator).

As illustrated in FIG. 1, in many embodiments, a thermal imaging device100 includes a controller or processor 102, non-volatile memory 103having an image correction system 114 and an image alignment system 116stored thereon, a non-thermal image sensor 104, a volatile memory module105, a thermal image sensor 106, any number of additional sensors 107, adisplay 108, a communication module 109, any number of inputs 110, andany number of outputs 112. In some forms, the thermal imaging device 100is contained in a single housing, as illustrated in FIGS. 2A and 2B. Inother examples, components of the device 100 are contained in multiplehousings. For example, some components such as the thermal image sensor106 and the controller 102 may be contained in a first housing, and thenon-thermal image sensor 104, display 108, and inputs 110 and outputs112 may be contained in a second housing. In these examples, a separatehandheld computing device such as a cellular telephone, tablet, or anyother device may be utilized. Components such as the non-thermal imagesensor 104, the controller 102, the display 108, and/or the inputs 110and outputs 112 may be a part of the handheld device, and the thermalimage sensor 106 and any other additional components may be adapted tobe coupled to and communicate with the handheld device using any numberof approaches known in the art. It is further understood that the device100 may include any number of additional components.

The controller 102 may be any computing device capable of reading andexecuting any number of computer-readable instructions and executearbitrary processing operations. The controller 102 may be a processoror processors, and may be in communication with any number of componentssuch as the volatile and non-volatile computer-readable storage modules103, 105 (e.g., random access memory and flash drives, hard drives,etc.), the wired and/or wireless communication modules 109, inputsand/or outputs 110, 112, and the like. These additional components maybe located on a common circuit board or may be located on anelectrically- or optically-connected circuit board. The controller 102is adapted to control operation of the thermal imaging sensor 106, thenon-thermal imaging sensor 104, the display 108, the image correctionsystem 114, the image alignment system 116, and any other additionalcomponents, as will be described in greater detail below.

The computer-readable instructions may be contained in the non-volatilestorage medium 103 or a separate connected non-volatile storage mediumand direct the processing circuitry to execute code providing variousfunctionality referred to as the operating system. Operating systems arecomponents of computing devices well-known to those skilled in the artand thus will not be discussed in further detail for the sake ofbrevity. The operating instructions may be complemented by astandard-based application programming interface (API) that provides anextensible model that may be expanded and reconfigured to perform anarbitrary sequence of application instructions even when the sequence ofprocessing is not presently known. The API provides abstractions ofvarious processing capabilities and functions. The functionalityprovided by software APIs is well-known to those skilled in the art, andthus will not be discussed in further detail.

The image correction system 114 and the image alignment system 116 maybe stored on the non-volatile memory module 103. In other embodiments,the image correction system 114 and the image alignment system 116 maybe located remotely from the device 100 (e.g., on a server), and maycommunicate with the controller 102 via any known communication methodusing the communication module 109. The image correction system 114 isadapted to identify reflections on assets in the environment. The imagealignment system 116 is adapted to align images captured in previousscans with either images currently being captured (or about to becaptured) or images captured in scans merely subsequent to the scanperformed by the operator of the thermal imaging system to generateimages taken from a uniform perspective to ensure the same portions ofthe asset are captured in each image.

The non-thermal image sensor 104 may be any type of visible, nearinfrared or other non-thermal wavelength light sensor. In oneembodiment, the non-thermal image sensor 104 may be a visible lightcamera which is sensitive to the visible light spectrum. The non-thermalimage sensor 104 is adapted to capture non-thermal light images and/orvideo of any number of assets in the environment. The thermal imagesensor 106 may be any type of imaging sensor adapted to capture thermalimages and/or video of any number of assets in the environment. Thethermal image sensor 106 may be any type of imaging sensor array whichis sensitive to thermal light such as, for example, a microbolometerarray, a thermopile array, a thermal diode array, or any other array ofsensors which are sensitive to photons and calibrated to produce anon-contact electronic measurement proportional to a temperature of anemissive asset in the thermal image array field of view. The non-thermalimage sensor 104 and/or the thermal image sensor 106 may transmit theimages and/or video to the controller 102 via a wired or wirelessconnection.

Non-thermal images and thermal images taken by the non-thermal imagesensor 104 and/or the thermal image sensor 106 may be combined in anumber of ways in order to enhance the information or interpretabilityof the individual images. Any of the visible-light images, thermalimages, or the combined images can be recorded for later examination,displayed on an output visible-light display system (e.g., the display108) on the device 100, or transferred to a separate display device(e.g., a separate computing device having a screen).

The display 108 may be any type of output visible-light display systemsuch as, for example, a liquid crystal display (LCD) or other displaydevice such as a TFT display. The display 108 may be coupled with atouchscreen interface for combined display and touchscreen user input.

The inputs 110 and outputs 112 may be any type of component orcomponents which allow the operator to interact with the device 100. Insome examples, the inputs 110 may be any number of buttons, switches,toggles, actuators, and the like which are coupled to the device 100. Insome forms, the inputs 110 may also be icons or buttons displayed on thedisplay 108. The outputs 112 may be coupled to the display 108 toprovide a visual representation of processing, analysis, or any othertask executed by the controller 102. Other examples of uses for theinputs 110 and/or outputs 112 are possible.

Sensor Calibration

As illustrated in FIG. 3, the non-thermal image sensor 104 generates afirst field of view 122 (denoted by forward slashes or “/”) and thethermal image sensor 106 generates a second field of view 124 (denotedby backwards slashes or “\”). The non-thermal image sensor 104 and thethermal image sensor 106 are arranged such that the fields of viewoverlap at area 126. This overlapping field of view 126 represents acombined imaging field of view, and may utilize computer-readableinstructions for performing a matched field of view transform and imagecombination operations when used with the non-thermal image sensor 104and thermal image sensor 106. A parallax offset and field of viewtransformation difference between the combined images can be correctedby applying a distance-dependent transform matrix.

Turning to FIGS. 4 and 5, in some examples, the controller 102 may beadapted to spatially calibrate the non-thermal image sensor 104 and thethermal image sensor 106 despite having different wavelengthsensitivities. In FIG. 5, visualization “A” depicts an image obtainedfrom the non-thermal sensor 104, visualization “B” depicts an imageobtained from the thermal image sensor 106, and visualizations “C” and“D” depict a combination of the images obtained from the non-thermalimage sensor 104 and the thermal image sensor 106. Other approaches forspatially calibrating the fields of view of the non-thermal image sensor104 and the thermal image sensor 106 are well known to those skilled inthe art.

In these examples, the environment can include structured elementsconsisting of a printed pattern (e.g., a checkerboard pattern 150)visible by the non-thermal image sensor 104. Any number of markers 152are placed at the corners of the pattern 150. These markers 152 mayeither be heated or cooled such that their temperature varies from theambient temperature sufficiently to be resolved from the background bythe thermal imaging sensor 106.

With reference still to FIGS. 4 and 5, at step 1 and step 4, thenon-thermal imaging sensor 104 and thermal imaging sensor 106 (havingoverlapping imaging fields of view) are directed at the scene such thatthe pattern 150 is within the image field of view of the non-thermalimaging sensor 104 (as illustrated in visualization A) and the thermallyvisible markers 152 are within the image field of view of the thermalimaging sensor 106 (as illustrated in visualization B). The visiblestructured elements (e.g., the elements viewable by the non-thermalimage sensor 104) are resolved within the visible field of view usingany number of approaches known in the art such as, for example, computervision software. At step 2, the visible image is processed to resolvethe visible image field of view pixel location of the inner corners ofthe pattern 150 on the visible image, and at step 5, the thermal pixellocation of the four heated or cooled markers 152 is resolved within thethermal image field of view using image thresholding, segmentation,spatial filtering, and/or averaging. Other approaches to resolving thethermal pixel locations are possible.

At step 3, and visualization C, a known transform is applied to theimage obtained from the thermal image sensor 106 to obtain an expectedlocation of the thermal markers 152 on the visible image using thecontroller 102. In other words, the non-thermal image field of view ismatched to the thermal image field of view. As shown in visualization C,the estimated location of the thermal markers illustrated in brokenlines does not match their location calculated from the visible camera.At step 6, an affine two-dimensional spatial transformation that matchesthese locations is performed by the controller 102. At step 7 andvisualization D, the affine transformation is applied to the thermalimage and results in a spatially calibrated thermal image matching thevisible image field of view. The matrix describing this affinecalibration is then stored for subsequent uses of the device as long asthe relative positioning or fixation between the non-thermal imagesensor 104 and the thermal image sensor 106 is unchanged during use.Accordingly, the non-thermal image sensor 104 and the thermal imagesensor 106 are properly calibrated to allow images from each sensor tobe aligned with each other.

It is understood that this process may be repeated for differentdistances between the sensing body and the imaging plane in order toproduce a set of distance-dependent calibration matrices. Thesedistance-dependent calibration matrices can be used to correct imagesacquired at a range of different distances having differing alignmentparallax offsets. In some examples, the operator is directed to select afixed distance before acquiring images. This distance can be used toselect a fixed image transform matrix to be used in all subsequent imageacquisition image processing to align the thermal sensor image with thevisual sensor image. In most cases, the image transform calibrationconstants are determined once for a particular spatial arrangement ofthe thermal image sensor and the non-thermal image sensor.

Image Alignment

In some examples, thermal and non-thermal images of an asset may bealigned in a uniform manner to allow for accurate analysis of changes tothe thermal profile of the asset over a specified time period. In theseexamples, an image is taken of the asset or an environment containingthe asset and is used as a template for future measurements of thatasset. This image or images may be non-thermal, thermal, both thermaland non-thermal, or a combination thereof, and may be captured by aprofessional thermographer. When a subsequent measurement of theparticular asset of interest is desired, the image alignment system 116,controlled by the controller 102, causes the template scene image to beshown on the display 108 linked to the imaging device 100.

The operator may then prepare to acquire images of the environment usingapproximately the same imaging orientation and distance from the assetas well as the same overall field of view of the environment, such thatthe environment matches the template environment as closely as possible.While acquiring the matching image, the original template environmentimage may be shown on the display 108 as a transparent and/ortranslucent overlay on the present image, the present image may beprocessed and displayed as a transparent and/or translucent overlay onthe template image, or only the template image or the present image maybe shown. In some examples, the template or present image may be furtherprocessed prior to display by first applying edge-detection imageprocessing (e.g., Canny edge detection) to the template image and/or thepresent image, and then overlaying the one processed image onto theother processed template or present image.

Numerous computational approaches are known for aligning images. Some ofthese approaches are classified as intensity-based alignment orfeature-based alignment. In these approaches, the alignmenttransformation between the images may be classified as either a linearor a nonlinear transformation. As a non-limiting example usingintensity-based image registration to determine a linear transform thataligns images, and as illustrated in FIG. 6, an approach 600 foraligning images taken at different times using an image alignment systemis described. There are numerous variations in methods for performingintensity-based alignment of images known to those skilled in the artincluding various optimization methods for adjusting the transformationsuch as, for example, markov random fields, New UnconstrainedOptimization Algorithm (NEWUOA) or gradient descent, various costfunctions for computing difference between transformed images or imagehistograms such, as but not limited to, least squares, mutualinformation, correlation ratio, local pearson correlation or normalizedcorrelation, various interpolation methods and various transformrestrictions such as, for example, rigid, partial affine with scale plusrigid or full affine.

In this example, baseline and current images (of the current field ofview) are acquired at 602, 604. The image alignment system 116 is thenapplied to the baseline image and/or the present image, and overlays thepresent image onto the template, baseline, or original image via gridresampling at step 606. At 608, a least squares cost function iscomputed of the sum of squares of the difference between the originaltemplate image and a processed version of the present image beingacquired. Image processing applied prior to the computation of the costfunction may include spatial transformation, histogram normalization,image segmentation, thresholding, spatial filtering, edge detection, orother such processing known by those in the art. The image differencecost function may be computed using a number of approaches such as, forexample, least-squares difference, normalized mutual information, localPearson correlation and the like. Other examples are possible.

The cost function may be displayed via the display 108 to assist theoperator in determining how close or far the present image is from theprevious image. In other examples, the cost function may be used afteracquiring many images to allow for post-hoc selection of an image orimages that most closely match the template scene, or the cost functionmay be used as a minimization function in an iterative search for alinear image transformation to spatially align the images such that theimages match the template more closely.

In one non-limiting example, the template image and the present imageare compared after processing by spatially transforming at least one ofthe images with a linear transformation matrix, by computing the sum ofsquares of the difference between the processed template and presentimages as the cost function. The cost function is then used to searchfor a linear image transform coregistration that minimizes 610 this costfunction. By searching for a transform matrix 612 that minimizes thecost function, the transformation that results in the best spatial matchbetween template and present images is determined. Many variations ofsearch strategy are possible, such as, for example, hierarchicalgradient descent. Further quality measures such as the residual leastsquares difference may be used to determine success of the registrationprocedure. It is understood that the image alignment system 116 mayapply image processing to a previously- and/or presently-capturednon-thermal and/or thermal image of the particular asset using thenon-thermal image sensor 104 and/or the thermal image sensor 106.

As a non-limiting example and as illustrated in FIG. 7, image featuresare used to align the images. In this example, the image alignmentsystem 116 computes and stores any number of image feature vectorsdescribing the environment to be recalled during subsequent scans. Thefeature descriptors can be computed using any known feature-baseddescription method, such as, for example, descriptors that are lesssensitive to camera pose and image intensity. Examples include, but arenot limited to, features based on a histogram of oriented gradients,oriented binary robust independent elementary features, binary robustinvariant scalable keypoints, fast retina keypoints and others.

The image alignment system 116 may determine feature vectors in thecurrent image and compare these vectors with the stored vectors usingapproaches known to those skilled in the art such as, for example,random sample, consensus (RANSAC), nearest neighbors, or brute forcematching. In the event of a correspondence above a threshold valuedetermined to indicate a match, the matched feature points from thecurrent image and the stored feature points may be passed to a methodfor determining the perspective transformation between matched pointsand the perspective transformation applied to the current image. If theperspective transform can be approximated by an affine homographytransform, then the perspective transform can be decomposed and usedsimilar to a linear rigid or affine transformation matrix. The resultingperspective-transformed current image may pass through a secondary imageco-registration procedure with the stored previously acquired image asthe registration target and the resulting image transformations may becombined with the perspective transform and the resulting transformapplied to the current non-thermal image data or paired thermal imagedata. The image alignment system 116 may then allow the previous thermaland/or non-thermal images to be compared with the present thermal and/ornon-thermal images to visualize and identify changes between the currentscene and the baseline scene using the display 108.

In some examples, the image alignment system 116 may invert thetransformation and apply the inverted transform to thepreviously-acquired images instead of the current image. In someexamples, the thermal level and span settings from the baseline orcurrent thermal image are applied to the current or baseline thermalimage while spatially matching the visual images, combining the visualto thermal sensor calibration transform with the visual scene transformfrom the image alignment system 116 and applying the derived transformor its inverse to one of the thermal images and processing both baselineand present images in order to display both at the same time blended orside-by-side. The baseline image may be the first image acquired of anasset, it may be updated periodically with contributions from laterimages, and/or it may be replaced by a later image, thus, it will beunderstood that the term “baseline image” can refer to a previouslyacquired image.

In some examples, the thermal image data obtained from one asset may becompared with thermal image data from a different, identical asset(e.g., two separate electrical panels). In these examples, theappearance of the two assets is sufficiently similar to allow a stableimage homography transform or image co-registration, the transform to beapplied to the second image, and images to be produced which appear tohave been taken from the same perspective relative to the similar asset.The operator may in turn directly compare thermal patterns of the twoassets by switching between the two images or performing a subtractionor other pixel-wise image transform to enhance the identification ofdifferences.

In other approaches, the image alignment system 116 includes an inertialmonitoring unit (IMU) coupled thereto. The IMU allows the cameraorientation with respect to ground to be recorded and can be used toannotate the base image's orientation. This annotation can be used toestimate the orientation component of the pose correction, which isbeneficial for later acquisition and computation by constraining theestimated pose to reduce errors and increase speed in computing the fullorientation plus translation pose correction.

The IMU orientation with respect to ground can also be used to guidesubsequent repeated image scene acquisition by feeding back thedifference between the current sensor orientation and the base sensororientation. It is noted that is not necessary that there be an IMUpresent for the full pose correction to be performed. During acquisitionof subsequent images, the user must maintain a suitable degree ofsimilarity in camera pose to the base camera pose. This similarity maybe enforced either using landmarks and knowledge of the scene to beacquired (e.g. an indicator on the floor denoting “stand here for assetA” or simply operator awareness of the scene and functioning of thepresent system), the base image display or base image transformed insome manner (as previously described), or a displayed estimated posecorrection between the current image to the base image can be used toguide the user. A real-time pose correction can be displayed insimplified form (e.g. which direction to move the camera center) inorder to guide the camera to match the current scene with the scene inthe representative base image. A transform between the current sceneimage and the base image can be generated through image registration,which may involve a perspective transform for all but small posedifferences, or may involve a linear transform particularly if the posedifference is small.

The results of a real-time image transform can be used to help guide theuser by assisting in the positioning of the camera, such as byindicating whether distance must be increased or decreased, or bydisplaying an indication signal proportional in some manner to the costfunction of the image transform. Furthermore, the guidance can besimplified and separated into independent degrees of freedom for ease inadjusting only one degree of freedom at a time, because adjustingmultiple degrees of freedom at once can be too difficult to perform. Theimage transform returned by the registration routine can be decomposedinto translation, rotation and scaling components. The scaling componentrelates to the difference in distance between the base and currentacquisition, thus a scaling factor greater or less than 1 indicates adifference in distance and a scaling factor of 1 means there is nodetectable difference in distance. Accordingly, scaling can be used toguide the user to the correct image acquisition distance.

As illustrated in FIG. 8, from the full pose correction or lineartransform, the translation and rotation can be used to direct theoperator to adjust the location and orientation of the camera in theplane perpendicular to the imaging axis such that it matches thelocation and orientation of the camera in the base camera pose.Alternatively, the orientation difference can be ignored if below asuitable threshold and any orientation differences can be handledpost-acquisition by a feature-based, intensity-based, or other imagealignment process performed by the image alignment system 116. Once thetotal transform difference from unity is below a pre-defined thresholdor error bound, the thermal and visible image or set of suitablybelow-error images are saved. Following acquisition, a finer-grained(higher accuracy) post-hoc co-registration of the acquired visible imageor images to the base visible image is computed, and applied to thevisible and thermal images, to more closely match the images to the baseimage. By synchronizing images on a pixel-by pixel basis, pixel-wisestatistical tests may be performed directly on the images.

In some examples, and as illustrated in FIG. 9, the device 100 mayutilize both feature-based alignment and orientation sensor-assistedalignment. In these examples, the image coregistration is decomposedinto an image scale component, a rotation component, and a translationcomponent. The components are used to display indicators on the display108 which correspond to an intended repositioning and re-orienting ofthe device 100 so as to match a present pose (or orientation) of thedevice 100 with a previous pose of the device 100. The orientationsystem may include sensors such as an accelerometer, an inclinometer, agyroscope, and/or an inertial monitoring unit, among other components.By measuring the pose of the system when capturing the previous orbaseline image of the asset, this information may be used in subsequentscans to initialize an image coregistration search routine, therebyreducing error and time in the computation.

Reflection Correction

In many embodiments, the image correction system 114 corrects thermalreflections in a scene acquired of an asset with the non-thermal imagesensor 104 and the thermal image sensor 106 by first eitherautomatically identifying thermal reflections or by a user manuallyindicating that a thermal reflection is expected to be present. Acorrection procedure to remove the effect of the thermal reflectionsincludes acquiring or guiding the user to acquire a sequence of thermaland/or non-thermal images of the particular asset with different posesand using the image alignment system 116 and any number of posecorrection techniques to produce a sequence of thermal images thatappear to have been acquired with the same pose. This same pose iseither a representative pose from one of the captured images or the mostrepresentative average (or median) pose of images acquired.

Upon acquiring these pose-corrected images, a sequence of thermal imagesare assembled for pixel-based correction. In one example, a statisticalfit to the pixel temperatures (such as, for example, the averagemeasured pixel temperature across pose-corrected acquired images) isused as the reflection-corrected pixel level. In other examples,additional information such as information from the discordance inoptical flow from the non-thermal and thermal images is used to furtherlimit the involvement of reflections from problematic temperaturedistributions in order to determine the reflection-corrected pixellevels. The reflection-corrected pixel level or levels can then beassociated with the reflected surface.

As illustrated in FIG. 10, potentially reflecting surfaces can beautomatically identified using a number of approaches such as, forexample, discordance in optical flow between the average or mostrepresentative optical flow within an image and individual regions orpixels of optical flow. Generally speaking, if a surface is reflectivein the visible or near infrared range, it is highly likely to bereflective in the thermal far infrared range. Thus, it is firstdetermined, using the image correction system 114, that datarepresentative of a reflected surface is present in a present thermalview of a particular asset. Features in the image or present view areidentified and the image alignment system 116 performs a homographydetection on the features between images acquired adjacent in timefollowed by the use of a random sample and consensus method which iswell known to those having skill in the art. The features identified andused for tracking give rise to optical flow vectors. In some examples,the optical flow vectors are analyzed for consistency. The mostconsistent optical flow vectors are removed and the procedure isrepeated to find the second most consistent pattern, if any. This isrepeated for adjacent optical flow detections and a second-passconsistency search is then performed. In the event that contiguousregions in space are identified with similar discordant optical flowpatterns from either the predicted optical flow from previousacquisitions or an IMU or a secondary optical flow pattern, a possiblereflector value is incremented. In general, no more than two opticalflow patterns will be searched for. In other examples, the optical flowvectors are used to determine an average or most representative vectoroptical flow as the global optical flow, and the local inconsistencybetween the local optical flow and the global optical flow is calculatedat each location as the optical flow error and an appropriate thresholdused to identify potential reflections. The optical flow error statisticcan be computed by taking the magnitude of the vector difference of thelocal optical flow with the global optical flow and dividing by thestandard deviation of all optical flow magnitudes or similar measureswithin the image. Thus, thermally reflective surfaces can be identifiedby first identifying non-thermally reflective surfaces. In some cases,surfaces that are not reflective in non-thermal wavelengths arethermally reflective and must be identified using other means. Surfacesthat are thermally reflective but are not non-thermally reflective canautomatically be identified using discordance in optical flow betweennon-thermal images and thermal images using the same or similarprocedures.

In some examples, the automatic detection of thermal reflections can beperformed by identifying any number of inconsistencies between imagemotion in the thermal and non-thermal images using image motionestimation techniques such as feature detection and optical flow, whichcomputes the vector motion of a set of individual pixels within animage. By assuming there are no or minimal non-thermal reflections, theoptical flow across the non-thermal image can be taken as a predictionof the optical flow expected to be observed in the thermal image. Atotal error between expected and measured optical flow is computed andused to indicate the presence of reflections. In some examples, a usermay manually input the presence of any number of reflections via inputs110.

In other examples, discordance between measured optical flow and apredicted optical flow can be used to identify the presence ofreflections. This measurement can be performed globally (e.g., theentire present view, image, or environment are assumed to be reflectingand a single image translation and rotation matrix are used) or locally(e.g., regions of the present view, image, environment, and/orparticular asset are assumed to be reflecting, thus, a model uses aprimary image translation and rotation matrix in addition to a secondarymatrix for the particular region.

Once the presence of reflections is indicated, whether manually orautomatically, a sequence of non-thermal and thermal image acquisitionsmust be made of the asset where the pose is varied over an adequatespace of possible poses by varying the angle of incidence, as measuredfrom the plane of the asset of the image acquisitions while the distanceof the image system from the center of the asset within the field ofview is held as constant as possible. Typically the pose should bevaried from some suitably shallow angle with respect to the asset, todeeper angles near perpendicular, to similar shallow angles but from theopposite side of the original center of the field of view, and to otherazimuthal angles about the asset, with a goal of suitable hemisphericimaging coverage. In some situations, there may be a restricted view ofthe asset which hinders the ability to fully cover an asset, in whichcase a smaller variety of angled poses with at least some coverage fromdirections opposite the axes of reflection. For example, as illustratedin FIG. 11, an operator can capture four images of an asset each from a45 degree angle of incidence. The sequence consists of an image takenfrom the left of, to the right of, from above and from below the asset,where the left image is opposite the reflective perspective of the rightimage, and the above image is the opposite of the below image. For eachacquisition, the homography between the non-thermal image and thenon-thermal image acquired under perpendicular pose or some mostrepresentative pose is determined by the image alignment system 116. Ifa suitably low-error homography transform cannot be performed, theacquisition is discarded, but if a suitably low-error homographytransform can be performed, this acquisition is retained and saved.

It should be noted that there is a limit to the acceptable shallownessor to the acceptable range of angles beyond which the perspectivecorrection fails to produce a pose-corrected image that is suitablysimilar to the baseline or most representative image, resulting in apartially hemispheric path of acquisition. It should also be noted thatthe desired view may not necessarily be at a 90 degree angle ofincidence, or perpendicular to the asset, but some other angle ofincidence chosen by the operator, such as 45 degrees from the plane ofthe asset. In other examples, a similarity measure or other qualitymetric is computed between each pose-corrected image and the baselineimage and a selected threshold is used to discard images that do notresult in a suitable pose-corrected image. When the pose-correctionmetric is too low, feedback can be provided to the user indicating theangle is too far from ideal and the user should adjust closer tobaseline or perpendicular or otherwise to a most representative angle.

This process is repeated for all poses, building a set of pose-correctednon-thermal and thermal images for further analysis. During theacquisition process, the optical flow for both thermal and non-thermalimages is determined, and the accumulated optical flow error for eachpose-corrected thermal image is stored. The storage and use of theoptical flow is optional, but as previously described, it can be used toexplicitly identify thermal reflections and mask their presence from thesequence of pose-corrected thermal images to produce a sequence ofpose-corrected and discordant flow-corrected thermal images.

Finally, at each pixel, the sequence of thermal values varies dependingon the pose and background features. In one example, the minimum thermalvalue can be used, assuming the surrounding space behind and around thedevice 100 does not contain any significantly colder surfaces than theambient temperature. Cold assets in the background can cause issues forthe automatic detection of a baseline if the cold asset will not bereflected into all pixels from the set of pose acquisitions attemptedbecause there will be some pose-corrected pixels in the scene that havecontributions from the cold asset, while other pixels may only havetheir coldest contribution from the average room surface temperature.There may be some other cold and emissive asset that could introduce lowvalues below the average room temperature, in which case this minimumtemperature across poses may not be desired and an alternativestatistical characteristic of said distribution used, such as theaverage. In most cases, the characteristic will be proportional tobackground or ambient temperature, and a measurement of the backgroundor ambient temperature can be recorded for the purpose of correcting themeasured reflection-corrected image to produce a repeatable measurementof reflective surfaces. Notably, images that have been corrected in sucha manner can be compared quantitatively, although the sensitivity tochanges in asset temperature at each pixel is proportional to the assetemissivity at each pixel. Below some level of emissivity, thesensitivity of this method may be insufficient to detect meaningfulchanges at some pixels, although the sensitivity may be enhanced in somecases using averaging techniques. This minimum emissivity will depend onnumerous factors including the sensitivity of the thermal sensor, thefield of view, angular resolution per pixel, distance to target anddistribution of asset and background temperature.

In alternative examples, a set of images of the background scenes thatare reflected by the asset is acquired, for example, by rotating thedevice 100 by 180 degrees and acquiring images of the scene that isbehind the operator when acquiring images of the asset. The distributionof thermal signatures observed are used to analyze the set ofmeasurements for a given pixel in the pose-corrected asset images suchthat the distribution of background emitters is matched to thedistribution of measurements observed at a given pixel and thereby canbe used to calculate a reflectivity coefficient and thereflection-corrected pixel value, repeated at each pixel. In analternate implementation, the cross-correlation of the set ofmeasurements for a given pixel and adjacent pixels is used to identifycommon reflection sources. The correlation phase shift is related to theangle of incidence and relative distance to foci of each pixel ofinterest. The component of maximal cross-correlation with adjacentpixels is used within a limited range of angles of incidence andrestricted in area to nearby pixels. The most common shape within alocal area is fitted by least-squares or partial correlation across aplurality of adjacent pixels to fit and remove sources of reflection.

The methods described herein for identifying reflections can be used in3-dimensional mapping from a sequence of images. In these examples,thermal images are generally not needed. Three-dimensional mappingprocedures that rely on images and feature-based registration are wellknown to those having skill in the art. As with other examples describedherein, a common problem in image-based 3-dimensional mapping is thepresence of reflective surfaces, which can result in image registrationsolutions that are incompatible with the 3-dimensional model. It isnoted that during the 3-dimensional mapping procedure, two incrementorscan be allocated for each location within the 3-dimensional model. Thefirst incrementor can refer to the number of image acquisitions with adetected reflection at the location while the other incrementor canrefer to the number of unique image acquisitions of the location. If thereflector value is greater than some threshold, such as, for example,50% of the number of unique image acquisitions of this location, then abinary decision is made that this particular location is considered areflection.

The predicted optical flow can be obtained from an extended Kalmanfilter or other temporally updating filter for the pose model orinterpolation between previous and subsequent pose, or can be obtainedby interpolating the average optical flow measured in previous orsubsequent images. When reflection methods are used in 3-dimensionalmapping, the 3-dimensional model of the surfaces of the spacesurrounding the device can include information used to calculateaccumulated optical flow error, which is the percent difference in theoptical flow versus a primary optical flow, accumulated confidence,which is the number of measurements in agreement in a 3-dimensionallocation below a given threshold of difference in location, and thereflectance of the surface or surfaces. In one example, the accumulatedoptical flow error is taken at each surface that has been previouslymapped and divided by the number of measurements of the surface. Athreshold is then used to identify excessive accumulated error, (suchas, for example, a threshold value of 0.5).

Those skilled in the art will recognize that a wide variety ofmodifications, alterations, and combinations can be made with respect tothe above described embodiments without departing from the scope of theinvention, and that such modifications, alterations, and combinationsare to be viewed as being within the ambit of the inventive concept.

What is claimed is:
 1. A thermal imaging system, the system comprising athermal imaging sensor configured to capture a plurality of thermalimages containing thermal data of a plurality of assets in anenvironment; a non-thermal imaging sensor coupled to the thermal sensor,the non-thermal imaging sensor configured to capture a plurality ofnon-thermal images of the plurality of assets; an image alignmentsystem; an image correction system; and a controller configured tocontrol operation of the thermal imaging sensor, the non-thermal imagingsensor, the image alignment system, and the image correction system by:(i) determining, using the image correction system, that datarepresentative of a thermally reflective surface is present in a presentthermal view of a particular asset by identifying at least one portionof the surface of the particular asset having thermal data correspondingto a location different than the at least one portion of the surface ofthe particular asset, wherein the at least one portion of the surface ofthe particular asset having thermal data corresponding to the locationdifferent than the at least one portion of the surface of the particularasset corresponds to the thermally reflective surface, (ii) causing thethermal imaging sensor to capture a plurality of thermal images and thenon-thermal imaging sensor to capture a plurality of non-thermal imagesof the particular asset at a plurality of poses, (iii) generating, usingthe image alignment system, a plurality of pose-corrected thermal imagesand a plurality of pose-corrected non-thermal images from the pluralityof thermal images and the plurality of non-thermal images, at least oneof the pose-corrected thermal images including corrected thermal datacorresponding to a location of the reflective surface, and (iv) usingthe image correction system and the plurality of thermal and non-thermalimages, associating the corrected thermal data corresponding to thelocation of the reflective surface with the at least one reflectivesurface present in the present thermal view.
 2. The thermal imagingsystem of claim 1, wherein the image correction system determines thatdata representative of a reflected surface is present by identifying aninconsistency between image motion in the captured thermal andnon-thermal images using an image motion estimation technique.
 3. Thethermal imaging system of claim 2, wherein the image motion estimationtechnique comprises at least one of a feature detection and an opticalflow detection that computes vector motion of a set of individual pixelswithin the captured plurality of thermal images.
 4. The thermal imagingsystem of claim 3, wherein the image alignment system determines ahomography between the present thermal view and the captured pluralityof thermal images.
 5. The thermal imaging system of claim 4, wherein theimage correction system determines an optical flow for each of theplurality of pose-corrected thermal images and computes an optical flowerror for each of the plurality of pose-corrected thermal images.
 6. Thethermal imaging system of claim 5, wherein the image correction systemuses a minimum thermal value calculated from the optical flow error toremove the thermal data corresponding to at least one reflection fromthe present thermal view.
 7. The thermal imaging system of claim 3,wherein the image alignment system determines a homography between thepresent non-thermal view and the plurality of pose-corrected non-thermalimages, and the image correction system determines an optical flow foreach of the plurality of pose-corrected non-thermal images and computesan accumulated optical flow error of the plurality of pose-correctednon-thermal images.
 8. The thermal imaging system of claim 1, whereinthe image correction system generates a statistic of pose correctedpixels and uses the statistic to replace the data representative of thereflective surface from the present thermal view.
 9. The thermal imagingsystem of claim 1, wherein the plurality of pose-corrected thermalimages and the plurality of pose-corrected non-thermal images arecaptured along a hemispherical path relative to the particular asset.10. The thermal imaging system of claim 1, wherein the image correctionsystem determines whether the at least one reflection is present byreceiving an input from a user.
 11. The thermal imaging system of claim1, wherein the controller is further configured to cause the non-thermalimaging sensor and the thermal imaging sensor to capture a plurality ofimages in an opposite direction from the particular asset to be used bythe image correction system.
 12. A method of correcting thermal imagesin an environment, the method comprising: obtaining at least one presentthermal view of at least one asset using a thermal imaging sensor and atleast one present non-thermal view of the at least one asset using anon-thermal imaging sensor; determining, using the image correctionsystem, that data representative of a thermally reflective surface ispresent in a present thermal view of a particular asset by identifyingat least one portion of the surface of the particular asset havingthermal data corresponding to a location different than the at least oneportion of the surface of the particular asset, wherein the at least oneportion of the surface of the particular asset having thermal datacorresponding to the location different than the at least one portion ofthe surface of the particular asset corresponds to the thermallyreflective surface; capturing a plurality of thermal and non-thermalimages of the at least one asset from a plurality of poses; generating,using the image alignment system, a plurality of pose-corrected thermalimages and a plurality of pose-corrected non-thermal images from theplurality of thermal images and the plurality of non-thermal images, atleast one of the pose-corrected thermal images including correctedthermal data corresponding to a location of the reflective surface;associating, using the image correction system and the plurality ofthermal and non-thermal images, the corrected thermal data correspondingto the location of the reflective surface with the at least onereflective surface present in the present thermal view.
 13. The methodof claim 12, wherein determining that data representative of areflective surface is present further comprises determining adiscordance between the present thermal view and the present non-thermalview.
 14. The method of claim 13, wherein determining the discordancecomprises at least one of performing a feature detection and an opticalflow detection.
 15. The method of claim 14, wherein adjusting theorientation of the plurality of thermal and non-thermal images comprisesdetermining a homography between the present thermal view and theplurality of captured thermal images.
 16. The method of claim 15,wherein determining the adjusted temperature value of the at least onereflection comprises determining an optical flow for each of theplurality of thermal images and computing an accumulated optical flowerror for each of the plurality of thermal images.
 17. The method ofclaim 12, wherein the image correction system generates a statistic ofpose corrected pixels and uses the statistic to replace the datarepresentative of the reflective surface from the present thermal view.18. The method of claim 12, wherein the plurality of thermal images andthe plurality of non-thermal images are captured along a hemisphericalpath relative to the particular asset.
 19. The method of claim 12,wherein identifying the reflection comprises manually notifying theimage correction system.
 20. The method of claim 12, further comprisingcapturing a plurality of thermal images and a plurality of non-thermalimages in an opposite direction from the particular asset to be used bythe image correction system.